Identification and evaluation of epidemic prediction and forecasting reporting guidelines: A systematic review and a call for action.

INTRODUCTION: High quality epidemic forecasting and prediction are critical to support response to local, regional and global infectious disease threats. Other fields of biomedical research use consensus reporting guidelines to ensure standardization and quality of research practice among researchers, and to provide a framework for end-users to interpret the validity of study results. The purpose of this study was to determine whether guidelines exist specifically for epidemic forecast and prediction publications. METHODS: We undertook a formal systematic review to identify and evaluate any published infectious disease epidemic forecasting and prediction reporting guidelines. This review leveraged a team of 18 investigators from US Government and academic sectors. RESULTS: A literature database search through May 26, 2019, identified 1467 publications (MEDLINE n = 584, EMBASE n = 883), and a grey-literature review identified a further 407 publications, yielding a total 1777 unique publications. A paired-reviewer system screened in 25 potentially eligible publications, of which two were ultimately deemed eligible. A qualitative review of these two published reporting guidelines indicated that neither were specific for epidemic forecasting and prediction, although they described reporting items which may be relevant to epidemic forecasting and prediction studies. CONCLUSIONS: This systematic review confirms that no specific guidelines have been published to standardize the reporting of epidemic forecasting and prediction studies. These findings underscore the need to develop such reporting guidelines in order to improve the transparency, quality and implementation of epidemic forecasting and prediction research in operational public health.

Author Correction: Global disease outbreaks associated with the 2015-2016 El Niño Event.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

A systematic review and evaluation of Zika virus forecasting and prediction research during a public health emergency of international concern.

INTRODUCTION: Epidemic forecasting and prediction tools have the potential to provide actionable information in the midst of emerging epidemics. While numerous predictive studies were published during the 2016-2017 Zika Virus (ZIKV) pandemic, it remains unknown how timely, reproducible, and actionable the information produced by these studies was. METHODS: To improve the functional use of mathematical modeling in support of future infectious disease outbreaks, we conducted a systematic review of all ZIKV prediction studies published during the recent ZIKV pandemic using the PRISMA guidelines. Using MEDLINE, EMBASE, and grey literature review, we identified studies that forecasted, predicted, or simulated ecological or epidemiological phenomena related to the Zika pandemic that were published as of March 01, 2017. Eligible studies underwent evaluation of objectives, data sources, methods, timeliness, reproducibility, accessibility, and clarity by independent reviewers. RESULTS: 2034 studies were identified, of which n = 73 met the eligibility criteria. Spatial spread, R0 (basic reproductive number), and epidemic dynamics were most commonly predicted, with few studies predicting Guillain-Barré Syndrome burden (4%), sexual transmission risk (4%), and intervention impact (4%). Most studies specifically examined populations in the Americas (52%), with few African-specific studies (4%). Case count (67%), vector (41%), and demographic data (37%) were the most common data sources. Real-time internet data and pathogen genomic information were used in 7% and 0% of studies, respectively, and social science and behavioral data were typically absent in modeling efforts. Deterministic models were favored over stochastic approaches. Forty percent of studies made model data entirely available, 29% provided all relevant model code, 43% presented uncertainty in all predictions, and 54% provided sufficient methodological detail to allow complete reproducibility. Fifty-one percent of predictions were published after the epidemic peak in the Americas. While the use of preprints improved the accessibility of ZIKV predictions by a median of 119 days sooner than journal publication dates, they were used in only 30% of studies. CONCLUSIONS: Many ZIKV predictions were published during the 2016-2017 pandemic. The accessibility, reproducibility, timeliness, and incorporation of uncertainty in these published predictions varied and indicates there is substantial room for improvement. To enhance the utility of analytical tools for outbreak response it is essential to improve the sharing of model data, code, and preprints for future outbreaks, epidemics, and pandemics.

Global Disease Outbreaks Associated with the 2015-2016 El Niño Event.

Interannual climate variability patterns associated with the El Niño-Southern Oscillation phenomenon result in climate and environmental anomaly conditions in specific regions worldwide that directly favor outbreaks and/or amplification of variety of diseases of public health concern including chikungunya, hantavirus, Rift Valley fever, cholera, plague, and Zika. We analyzed patterns of some disease outbreaks during the strong 2015-2016 El Niño event in relation to climate anomalies derived from satellite measurements. Disease outbreaks in multiple El Niño-connected regions worldwide (including Southeast Asia, Tanzania, western US, and Brazil) followed shifts in rainfall, temperature, and vegetation in which both drought and flooding occurred in excess (14-81% precipitation departures from normal). These shifts favored ecological conditions appropriate for pathogens and their vectors to emerge and propagate clusters of diseases activity in these regions. Our analysis indicates that intensity of disease activity in some ENSO-teleconnected regions were approximately 2.5-28% higher during years with El Niño events than those without. Plague in Colorado and New Mexico as well as cholera in Tanzania were significantly associated with above normal rainfall (p < 0.05); while dengue in Brazil and southeast Asia were significantly associated with above normal land surface temperature (p < 0.05). Routine and ongoing global satellite monitoring of key climate variable anomalies calibrated to specific regions could identify regions at risk for emergence and propagation of disease vectors. Such information can provide sufficient lead-time for outbreak prevention and potentially reduce the burden and spread of ecologically coupled diseases.

Predicting influenza with dynamical methods.

BACKGROUND: Prediction of influenza weeks in advance can be a useful tool in the management of cases and in the early recognition of pandemic influenza seasons. METHODS: This study explores the prediction of influenza-like-illness incidence using both epidemiological and climate data. It uses Lorenz's well-known Method of Analogues, but with two novel improvements. Firstly, it determines internal parameters using the implicit near-neighbor distances in the data, and secondly, it employs climate data (mean dew point) to screen analogue near-neighbors and capture the hidden dynamics of disease spread. RESULTS: These improvements result in the ability to forecast, four weeks in advance, the total number of cases and the incidence at the peak with increased accuracy. In most locations the total number of cases per year and the incidence at the peak are forecast with less than 15 % root-mean-square (RMS) Error, and in some locations with less than 10 % RMS Error. CONCLUSIONS: The use of additional variables that contribute to the dynamics of influenza spread can greatly improve prediction accuracy.

Exposure science in an age of rapidly changing climate: challenges and opportunities.

Climate change is anticipated to alter the production, use, release, and fate of environmental chemicals, likely leading to increased uncertainty in exposure and human health risk predictions. Exposure science provides a key connection between changes in climate and associated health outcomes. The theme of the 2015 Annual Meeting of the International Society of Exposure Science-Exposures in an Evolving Environment-brought this issue to the fore. By directing attention to questions that may affect society in profound ways, exposure scientists have an opportunity to conduct "consequential science"-doing science that matters, using our tools for the greater good and to answer key policy questions, and identifying causes leading to implementation of solutions. Understanding the implications of changing exposures on public health may be one of the most consequential areas of study in which exposure scientists could currently be engaged. In this paper, we use a series of case studies to identify exposure data gaps and research paths that will enable us to capture the information necessary for understanding climate change-related human exposures and consequent health impacts. We hope that paper will focus attention on under-developed areas of exposure science that will likely have broad implications for public health.

Make Data Sharing Routine to Prepare for Public Health Emergencies.

Jean-Paul Chretien and colleagues argue that recent Ebola and Zika virus outbreaks highlight the importance of data sharing in scientific research.

Adapting to Health Impacts of Climate Change in the Department of Defense.

The Department of Defense (DoD) recognizes climate change as a threat to its mission and recently issued policy to implement climate change adaptation measures. However, the DoD has not conducted a comprehensive assessment of health-related climate change effects. To catalyze the needed assessment--a first step toward a comprehensive DoD climate change adaptation plan for health--this article discusses the DoD relevance of 3 selected climate change impacts: heat injuries, vector-borne diseases, and extreme weather that could lead to natural disasters. The author uses these examples to propose a comprehensive approach to planning for health-related climate change impacts in the DoD.

Prediction of Peaks of Seasonal Influenza in Military Health-Care Data.

Influenza is a highly contagious disease that causes seasonal epidemics with significant morbidity and mortality. The ability to predict influenza peak several weeks in advance would allow for timely preventive public health planning and interventions to be used to mitigate these outbreaks. Because influenza may also impact the operational readiness of active duty personnel, the US military places a high priority on surveillance and preparedness for seasonal outbreaks. A method for creating models for predicting peak influenza visits per total health-care visits (ie, activity) weeks in advance has been developed using advanced data mining techniques on disparate epidemiological and environmental data. The model results are presented and compared with those of other popular data mining classifiers. By rigorously testing the model on data not used in its development, it is shown that this technique can predict the week of highest influenza activity for a specific region with overall better accuracy than other methods examined in this article.

Mathematical modeling of the West Africa Ebola epidemic.

As of November 2015, the Ebola virus disease (EVD) epidemic that began in West Africa in late 2013 is waning. The human toll includes more than 28,000 EVD cases and 11,000 deaths in Guinea, Liberia, and Sierra Leone, the most heavily-affected countries. We reviewed 66 mathematical modeling studies of the EVD epidemic published in the peer-reviewed literature to assess the key uncertainties models addressed, data used for modeling, public sharing of data and results, and model performance. Based on the review, we suggest steps to improve the use of modeling in future public health emergencies.

Global climate anomalies and potential infectious disease risks: 2014-2015.

BACKGROUND: The El Niño/Southern Oscillation (ENSO) is a global climate phenomenon that impacts human infectious disease risk worldwide through droughts, floods, and other climate extremes. Throughout summer and fall 2014 and winter 2015, El Niño Watch, issued by the US National Oceanic and Atmospheric Administration, assessed likely El Niño development during the Northern Hemisphere fall and winter, persisting into spring 2015. METHODS: We identified geographic regions where environmental conditions may increase infectious disease transmission if the predicted El Niño occurs using El Niño indicators (Sea Surface Temperature [SST], Outgoing Longwave Radiation [OLR], and rainfall anomalies) and literature review of El Niño-infectious disease associations. RESULTS: SSTs in the equatorial Pacific and western Indian Oceans were anomalously elevated during August-October 2014, consistent with a developing weak El Niño event. Teleconnections with local climate is evident in global precipitation patterns, with positive OLR anomalies (drier than average conditions) across Indonesia and coastal southeast Asia, and negative anomalies across northern China, the western Indian Ocean, central Asia, north-central and northeast Africa, Mexico/Central America, the southwestern United States, and the northeastern and southwestern tropical Pacific. Persistence of these conditions could produce environmental settings conducive to increased transmission of cholera, dengue, malaria, Rift Valley fever, and other infectious diseases in regional hotspots as during previous El Niño events. DISCUSSION AND CONCLUSIONS: The current development of weak El Niño conditions may have significant potential implications for global public health in winter 2014-spring 2015. Enhanced surveillance and other preparedness measures in predicted infectious disease hotspots could mitigate health impacts.

Expectation management.

As physicians transform observed frequencies from studies into predicted probabilities for a given patient, we generally fail to consider that the predictions we utter about a given therapeutic intervention are themselves part of the intervention.

A review of evaluations of electronic event-based biosurveillance systems.

Electronic event-based biosurveillance systems (EEBS's) that use near real-time information from the internet are an increasingly important source of epidemiologic intelligence. However, there has not been a systematic assessment of EEBS evaluations, which could identify key uncertainties about current systems and guide EEBS development to most effectively exploit web-based information for biosurveillance. To conduct this assessment, we searched PubMed and Google Scholar to identify peer-reviewed evaluations of EEBS's. We included EEBS's that use publicly available internet information sources, cover events that are relevant to human health, and have global scope. To assess the publications using a common framework, we constructed a list of 17 EEBS attributes from published guidelines for evaluating health surveillance systems. We identified 11 EEBS's and 20 evaluations of these EEBS's. The number of published evaluations per EEBS ranged from 1 (Gen-Db, GODsN, MiTAP) to 8 (GPHIN, HealthMap). The median number of evaluation variables assessed per EEBS was 8 (range, 3-15). Ten published evaluations contained quantitative assessments of at least one key variable. No evaluations examined usefulness by identifying specific public health decisions, actions, or outcomes resulting from EEBS outputs. Future EEBS assessments should identify and discuss critical indicators of public health utility, especially the impact of EEBS's on public health response.

Influenza forecasting in human populations: a scoping review.

Forecasts of influenza activity in human populations could help guide key preparedness tasks. We conducted a scoping review to characterize these methodological approaches and identify research gaps. Adapting the PRISMA methodology for systematic reviews, we searched PubMed, CINAHL, Project Euclid, and Cochrane Database of Systematic Reviews for publications in English since January 1, 2000 using the terms "influenza AND (forecast* OR predict*)", excluding studies that did not validate forecasts against independent data or incorporate influenza-related surveillance data from the season or pandemic for which the forecasts were applied. We included 35 publications describing population-based (N = 27), medical facility-based (N = 4), and regional or global pandemic spread (N = 4) forecasts. They included areas of North America (N = 15), Europe (N = 14), and/or Asia-Pacific region (N = 4), or had global scope (N = 3). Forecasting models were statistical (N = 18) or epidemiological (N = 17). Five studies used data assimilation methods to update forecasts with new surveillance data. Models used virological (N = 14), syndromic (N = 13), meteorological (N = 6), internet search query (N = 4), and/or other surveillance data as inputs. Forecasting outcomes and validation metrics varied widely. Two studies compared distinct modeling approaches using common data, 2 assessed model calibration, and 1 systematically incorporated expert input. Of the 17 studies using epidemiological models, 8 included sensitivity analysis. This review suggests need for use of good practices in influenza forecasting (e.g., sensitivity analysis); direct comparisons of diverse approaches; assessment of model calibration; integration of subjective expert input; operational research in pilot, real-world applications; and improved mutual understanding among modelers and public health officials.